https://issues.apache.org/jira/browse/ARROW-1614

Currently, only the shape is stored. Is this enough? That does a assume a fixed row major order?

Currently, only the shape is stored. Is this enough? That does a assume a fixed row major order?

I think we either assume that or also store strides / dimension order. I am not sure how dimension order changes are done in other frameworks (TF, pytorch, etc.) but I would assume they don't reorder tensors in memory. So I would go for storing strides.

In the context of testing metadata equality withinin multiple parquet files in a dataset, equality on shape and strides may be a very strict requirement. Would relaxing the equality requirement to only compare the number of tensor dimensions negatively impact the design?

In the context of testing metadata equality withinin multiple parquet files in a dataset, equality on shape and strides may be a very strict requirement. Would relaxing the equality requirement to only compare the number of tensor dimensions negatively impact the design?

Good point. By tensor dimensions you mean shape, right?
I think it's ok to relax on the strides check. I've pushed a change, see latest commit.

In the context of testing metadata equality withinin multiple parquet files in a dataset, equality on shape and strides may be a very strict requirement. Would relaxing the equality requirement to only compare the number of tensor dimensions negatively impact the design?

Good point. By tensor dimensions you mean shape, right? I think it's ok to relax on the strides check. I've pushed a change, see latest commit.

I was thinking even looser:

def __eq__(self, other):
    len(self.shape) == len(other.shape)

I was thinking even looser:

def __eq__(self, other):
    len(self.shape) == len(other.shape)

Done.
We could introduce comparison options in case there would be differing requirements here.

@jorisvandenbossche @sjperkins @pitrou is there interest to get this in?
If yes is cpp/src/arrow/extension_type_test.cc a good place to put it?

Currently we don't ship any standard extension types. I recommend discussing this on the mailing-list.

fyi, the ray project created its own Tensor type:
https://docs.ray.io/en/latest/_modules/ray/data/extensions/tensor_extension.html#ArrowTensorArray

Indeed I think having a built-in Tensor value type (implemented using extension arrays) in Arrow/pyarrow would be better than having third party projects rolling their own.

@wesm would there be interest in folding the Pandas side of these third-party extensions into Pandas also?

would there be interest in folding the Pandas side of these third-party extensions into Pandas also?

That will be something to discuss in the pandas project.
(speaking as a pandas maintainer, for now we mostly encourage creating third-party extensions (that was the whole purpose of formalizing this ExtensionArray interface in pandas), but at some point we should also expand the types in pandas itself. Although I personally think we should rather start with adding a simple List type, than directly a tensor type)

Thanks for the fast review @jorisvandenbossche. I've addressed your points.

@jorisvandenbossche it seems CI is satisfied.

@rok would it be possible to add a method for value_type similar to shape(), permutation() and dim_names()? I think I can not currently retrieve it in Python.

@AlenkaF sure, I'll add it.

@AlenkaF done.

@rok, you are awesome! 👍

Great news!

The failing CI is unrelated? (it seems the R failures are being worked on, and the C++ failures are related to LLVM update #34768)

They seem unrelated indeed and I don't think they obscure any new problems as the change was fairly minimal.

Merged after 2.5 years ;) Thanks @rok!

Thanks for all the input and reviews everyone, very happy to see this merged!

@jorisvandenbossche now let's talk about strides @ #34797 :D

Benchmark runs are scheduled for baseline = 81c828e and contender = a84a39b. a84a39b is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.56% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.26% ⬆️0.0%] ursa-i9-9960x
[Failed ⬇️0.0% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] a84a39b6 ec2-t3-xlarge-us-east-2
[Failed] a84a39b6 test-mac-arm
[Finished] a84a39b6 ursa-i9-9960x
[Failed] a84a39b6 ursa-thinkcentre-m75q
[Finished] 81c828ed ec2-t3-xlarge-us-east-2
[Failed] 81c828ed test-mac-arm
[Finished] 81c828ed ursa-i9-9960x
[Failed] 81c828ed ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

We started a mailing list discussion about potential VariableShapeTensor extension array, please check it out and give feedback. For more details here's also a PR #37166.